Interference-aware Scheduling for Data-processing Frameworks in Container-based Clusters

With the emergence of data-processing frameworks like Hadoop and Spark, a new concept of a cluster resource manager was necessary to deliver per-application containerwrapped on-demand resources with high scalability on a large scale. The container-based ”Big Data” Operating System concept arose (e.g. YARN and Mesos) and brought along with it the long-standing inter-instance performance interference issues from virtualization technologies. As a result, the performance interference effects between co-located data-processing applications become an uncertain issue in container-based clusters, leading performance to fluctuate unpredictably and the guarantees to be likely violated. To work around this, an interference-aware scheduling algorithm is necessary to mitigate interference-related performance degradation and wisely schedule tasks on the bestsuited compute nodes—the nodes whose performance is maximized and the makespan is minimized.